chore(streaming): sync #784 with main as a true merge, restoring main's ancestry - #815

Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@dbcb813...81c6f93)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…Catalog (#785)

* feat(databricks): column restrictions and row filters compile to Unity Catalog SQL functions

policy.authz.columnRestrictions and rowFilters were refused. They now compile to SQL
functions named by a hash of their definition, with the semantics GCP and AWS give the same
fields: deny is a mask that returns NULL, mask is last_four, first_four or NULL, one mask per
column with a deny beating a mask, and one row filter per table that is the union of its
principals' predicates for their members and every row for everyone else. A predicate is
parsed as one boolean expression, checked node by node against an allowlist and rendered
from the tree. access: allow, a principal SQL cannot test and a text mask on a non-string
column are refused. Nothing runs them yet.

* feat(databricks): fluid apply binds the masks and row filter around tofu apply, and revokes SELECT when it cannot

The provider has no function resource and no mask or filter attribute, so the statements run
on the SQL warehouse through the SQL Statement Execution API. pre_apply binds on the tables
that exist before tofu apply runs, so a grant the apply adds lands on a protected table;
post_apply binds on the tables the apply created, drops a mask or filter this module bound
that the contract no longer asks for, and, for a table it cannot bind, revokes SELECT from
the principals the contract restricts and exits non-zero. Live bindings are read from the
Tables API and only what differs runs, so a second apply executes nothing. A dry run prints
the statements and runs none. The stand-in executes the statements and serves the bindings.

* feat(verify): fluid verify checks a Databricks table's row filter and column masks

The contract's restrictions are Unity Catalog functions bound on the table, named by a hash
of their definition. Verify compares the table's live row_filter and each column's mask,
from the GetTable it already reads, with the names the contract expects: a binding that is
missing, or is another function (a changed predicate is a new name), is CRITICAL and the fix
is fluid apply. No restriction in the contract skips the dimension.

* test(databricks): a real tofu apply binds the masks and row filter against the stand-in, and fails safe

Apply binds the filter and masks and a second apply executes no statement; a table that
exists is bound before the apply can grant on it; a changed predicate binds a new function
and keeps the old; a removed restriction is dropped and said to widen access; a binding
that fails revokes SELECT from the restricted principals only, fails the run, and the next
apply binds first and grants again; a failure before the apply stops it with nothing applied.

* docs(databricks): the changelog and tested-honestly rows say what is enforced, how, and what is not proven live

* fix(databricks): a contract with no restriction needs no databricks-sdk at apply and says nothing without it

* fix(databricks): a mask or filter tests every identity as a group and as a user, so a remapped principal is still restricted

The test was chosen from the logical principal's prefix. binding.principals may map a group:
principal to a service principal's application ID or a user's email, and the group test then
matched no one while the grant to that identity still applied: the mask and filter never
took effect. Every identity is tested as is_account_group_member(x) OR current_user() = x;
a name that is both a group and a user is restricted either way, which errs closed. The
prefix only refuses domain: and unprefixed principals now.

* fix(databricks): a mask rule belongs to the principal that names it, and the strictest matching one applies

One rules[column] kept the last restriction's rule for every masked member, so a nullify for
one principal and a last_four for another gave the first the last four characters, a weaker
mask than declared. Members are grouped by rule and the function has one WHEN branch per
rule, strictest first: deny, nullify, first_four, last_four, so a member of several groups
gets the strictest that matches. Each text rule is type-checked against the column.
restrictions_for takes rule_per_principal for a platform whose mask tests each principal
itself; GCP and AWS keep refusing two rules for one column.

* fix(databricks): a tofu apply that fails part way still binds the masks and row filter of the tables that exist

A failed apply raised before post_apply ran, so a table it created, its grants and the dbt job
that fills it could exist with no mask or filter, and a scheduled run would expose unmasked
rows. Before the apply error is raised, post_apply now runs best effort in a bind-only mode
(after_failed_apply): it binds what exists, revokes SELECT where it cannot bind, skips a table
that does not exist, and drops nothing, since dropping widens access and waits for an apply
that succeeded. A failure there is printed; the run still raises opentofu_apply_failed.

* docs(databricks): the changelog and tested-honestly rows describe per-identity tests, per-principal mask rules and the bind-only pass after a failed apply

* fix(databricks): a function under a hashed name is read back and must be the one this module creates, before it is bound

The name fluid_mask_<table>_<column>_<hash8> can be computed from the contract, so anyone with
CREATE FUNCTION on the schema could create it first with a body that returns the value
unmasked: CREATE FUNCTION IF NOT EXISTS kept theirs, SET MASK bound it and fluid verify, which
compared names only, passed. After each create and before any bind the function is read
(GET /api/2.1/unity-catalog/functions/{name}) and its owner, body, parameters and return type
must be the expected ones, whitespace-normalised; the owner must be the identity the run
authenticates as (userName of scim/v2/Me, who CREATE FUNCTION makes the owner). A mismatch is a
failed bind, printed with what differs. fluid verify reads every bound function back and checks
the same, accepting the table's owner as well since verify may run as someone else.

* fix(databricks): a row filter or mask set by hand is never replaced silently

SET ROW FILTER and SET MASK overwrite whatever is bound, so a filter or mask an admin set (a
function without the fluid_ prefix) was lost without a word when the contract asked for its
own. Every table is read and planned before any statement runs, and a live binding this
module did not create, where the contract asks for one, is refused naming the table, the
column and the live function, with the remedy: remove it, or move its rule into the contract.
Nothing is changed on any table. After an apply it is a failed bind (the fail-safe runs).

* fix(databricks): a Tables API read that fails for one table is a failed bind for it, and the other tables are still tried

_read raised outside the try that triggers the fail-safe, so a transient 429 or 503 on one
table aborted post_apply with that table granted and unbound and the rest never tried. Read
failures are caught per table, after an apply and after a failed one: a table with
restrictions is a failed bind (the fail-safe runs, the error names it) and the loop goes on;
one without is a note. A read that fails before the apply still stops the run.

* fix(databricks): a failed bind binds a deny-all row filter first, and only then locks the table by revoking

The fail-safe revoked SELECT from the restricted principals alone, which misses a user in
another granted group and SELECT inherited from the schema or catalog, and the message claimed
none of them could read unmasked rows. A failed bind now binds a fixed-purpose deny-all row
filter for the restricted identities (a zero-column function, RETURN NOT <any restricted
identity>, content-hashed and checked for owner and body like every function): it blocks them
whatever the grant path. If that cannot be bound, or would replace a row filter this module
did not create, SELECT (and ALL PRIVILEGES) is revoked on the table from every grantee but its
owner, listed through the permissions API. The message says exactly what was done and names
each grant path still open, a schema or catalog grant a table-level REVOKE cannot close, with
the statement to run.

* fix(databricks): a row filter that needs a column the apply adds is held by a deny-all filter until then

pre_apply skipped a filter whose columns the table does not have yet, and the table exists
with rows, so the restricted principals read every row until post_apply. When the real filter
cannot bind and the table exists, pre_apply now binds the interim deny-all row filter (the
fail-safe's own: a zero-column function, SET ROW FILTER ... ON (), owner and body checked)
for the restricted identities, and post_apply replaces it with the real one. A dry run shows
it; a hostile function under its name, or a row filter this module did not create, stops the
run before the apply. SET replaces a bound filter directly: a DROP first would leave a window
with no filter.

* fix(databricks): generate iac refuses a module a plain tofu apply would leave unmasked

The masks and row filters are bound on the SQL warehouse around tofu apply, not in the module,
so fluid generate iac wrote a module that creates the table and grants SELECT with none of
them and told the reader to review and apply it with OpenTofu. generate iac now asks the
plugin (standalone_module_refusal) before it writes one, and refuses a Databricks contract
that compiles any mask or row filter as databricks-enforcement-outside-module, saying that a
plain tofu apply would grant unmasked access and to use fluid apply. emit stays pure; fluid
apply, plan and validate are unaffected.

* fix(databricks): a dbt job that writes a table not yet protected is created paused

A continuous or frequent cron job created in the same tofu apply as its restricted table and
the grant on it could run and fill the table before post_apply binds the masks and row filter.
DatabricksIacPlugin.emit (the apply path; the pure compile validate uses is unchanged) now
reads the live bindings of each restricted table a job writes and sets pause_status to PAUSED on
the job's schedule or continuous block unless the table exists, its row filter and masks are the
expected functions on the expected columns, and each function is the one this module creates.
An unreadable workspace counts as not protected. The next fluid apply, once they are bound,
unpauses it in place, with no drift from unpausing by hand. A manual job has nothing to pause.
plan_notices says why a job is created paused and that run-now works meanwhile. The engine
comment that claimed binding happens before any build can fill a table is corrected.

* docs(databricks): the changelog and tested-honestly rows describe the verified functions, the deny-all fail-safe, the paused jobs and what is still unproven live

* chore(tests): the unused names the code-quality review found are gone

An unused import of copy and Optional, two unused helper imports and an
unused unpacking target in the Databricks tests; the stand-in's deploy
identity is now the one constant the module itself uses.

Bumps [reviewdog/action-actionlint](https://github.com/reviewdog/action-actionlint) from 1.80.6 to 1.81.1.
- [Release notes](https://github.com/reviewdog/action-actionlint/releases)
- [Commits](reviewdog/action-actionlint@48e36f5...619a81e)

---
updated-dependencies:
- dependency-name: reviewdog/action-actionlint
  dependency-version: 1.81.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…her (#745)

Dependabot split the 4.39.1 bump into #745 (init) and #747 (analyze). CodeQL fails any run whose steps are on different versions, so each PR alone was red on all three Analyze jobs. This one now bumps both to the same pinned SHA (9b3254085494b55a0f7c4c2b3d2d52cc712ce370, checked against the upstream v4.39.1 tag), and dependabot.yml groups github/codeql-action* so init, analyze and upload-sarif arrive in one PR from now on.

…nTofu apply failure (#786)

* feat(doctor): --strict exits 1 when doctor reports "Action needed"

fluid doctor printed "Action needed" for a copilot that is not ready and
still exited 0, so a CI gate on doctor could not fail. With --strict it
exits 1 whenever the summary says "Action needed", in the text report and
with --json, whose payload now carries action_needed and copilot_ready.
The default exit code is unchanged; --strict has no effect with
--features-only, which prints no summary.

Found by an external hands-on assessment of 0.26.0.

* fix(roadmap): stop claiming a 1.0 release; name every provider in the description

fluid roadmap printed "forge-cli v1.0 shipped 2026-04-23" while the
package is 0.x and classified Beta. The roadmap now says so and calls its
"v1" headings an internal milestone, and a test fails if it announces a
1.x release while the package is below 1.0. The package description named
GCP, Snowflake and Local; it now also names AWS and Databricks.

Found by an external hands-on assessment of 0.26.0.

* fix(apply): a failed OpenTofu apply names its diagnostic and the way on, redacted

The runner uses -json, so the error a failed apply printed was the last
800 characters of diagnostic JSON lines, with no hint. The error now reads
the first error diagnostic (summary, resource address, provider detail),
falls back to the first "Error:" or the exit code, and says that OpenTofu
records state as it goes, so fixing the cause and re-running fluid apply
normally continues; it names the workdir for `tofu state list`, and only a
state-lock error points at `tofu force-unlock`. The raw tail goes to
--debug.

Every value a CLIError prints is now redacted and has its control
characters spelled out, as the other error handlers already did: the new
detail field carries provider text, which can hold a credential.

Found by an external hands-on assessment of 0.26.0.

* fix(cli): error values go through the full secret redactor; a locked plan says how to clear it

The top-level CLIError handler ran context values through _console's
narrow pattern only (bare password/token/secret/api key followed by : or
=), which leaves an AKIA access key id, a Bearer token, a URL password,
JSON-quoted and *_secret_key shapes and env-literal secrets in clear.
Context values and suggestions now go through redact_secret_text first,
the redactor the logs already use. _apply_failure redacts the diagnostic
before cutting it at 800 characters, since a credential split at the
limit (a URL that lost its @) matches no pattern.

A stale state lock first surfaces at tofu plan, which printed only the
raw tail; _plan_failure now redacts it and, for a state-lock error only,
says to wait for the other run or clear it with tofu force-unlock in the
named workdir.

Found by the security review of this branch.

* test: allowlist the deliberate fake credentials the secret scan flags

detect-secrets failed Lint & Format on the AWS-example key and the
admin:s3cretpw DSN these redaction tests feed in on purpose. Each value
now carries `# pragma: allowlist secret` on every line that holds it,
not just the first: detect-secrets reports one finding per value per
file, so allowlisting only lines 690/696 and 412/413 would have surfaced
the copies at 695/701 and 424 on the next run.

Two lines were pre-split so the pragma stays on the value. With the
comment appended, black would otherwise rewrap them and leave the pragma
on a closing bracket, where detect-secrets does not look.

…792)

run_welcome_scan drops any probe result that misses its 0.4s/1.0s budget,
and pick_mode takes its default from those results. On a loaded runner the
workspace probe overran and test_pick_mode_default_picks_from_product_when_
workspace_has_products saw "ai" instead of "from_product" (PR #782, Python
3.14 coverage leg).

Add a deterministic_welcome_scan fixture that lifts the budgets and points
the dbt probe at "no dbt" (it sizes its `dbt --version` subprocess from the
same budget), and apply it to the four modules whose tests assert on probe
results. Exposure audit: with every probe result dropped, 13 tests in those
four modules fail; the other test files that reach the scan do not. A new
test pins the original failure with a deliberately slow workspace probe.

The product budget is unchanged.

…788)

* security: validate_ident let a name with a trailing newline through

Python's `$` also matches just before a trailing newline, so
`_SAFE_IDENT.match("orders\n")` succeeded and validate_ident returned
"orders\n" unchanged. Every DDL emitter routes identifiers through it,
and a name written into a line-based file (a Java properties file, as
in #784) gains a second line.

The four whole-string allowlists in _sql_safety.py now end in `\Z`.
Only _SAFE_IDENT changes behaviour: the type-name, type-parameter and
RLS-expression helpers already strip their input, so `\Z` there keeps
a later edit that drops the strip from reopening the gap.

tests/providers/test_sql_safety_type_param.py pins validate_ident
against "\n", "\r\n" and an embedded "\n", and each pattern against a
trailing "\n".

* docs(changelog): validate_ident refuses a name with a trailing newline

* chore(secrets): move the two _sql_safety baseline hits to their new lines

* ci: freeze dependency weight and startup cost before the diet

Guards at today's numbers, so nothing grows while later PRs move litellm,
mcp and openlineage out of the base install.

- scripts/ci/dep_budget.py and .github/workflows/dep-budget.yml resolve
  the base install with `uv pip compile` for linux/3.12, linux/3.10 and
  windows/3.12 and fail above a committed ceiling (88/86/89), naming the
  packages added and removed against a baseline. It is its own workflow
  (PRs touching pyproject, main, weekly), because an upstream release can
  move the count with no change here.
- scripts/ci/bare_install_smoke.sh, called from ci.yml build-check and
  release.yml, caps the bare wheel's build_parser() import cost (869) and
  runs validate, plan, bundle, generate iac and doctor on an AWS example.
- .importlinter-external is an import-linter forbidden contract with
  external packages: litellm, mcp, openlineage and boto3 stay off the core
  modules. The 8 lazy routes that exist today are listed one by one.
- ruff TID253 bans module-level imports of the heavy SDKs, with 4
  exemptions. It does not see an import inside a module-level try:, which
  the config says.
- deptry 0.25.1 joins [dev]. CI runs it as advisory until 0.29.0.
- tests/perf/test_startup_budget.py lowers MAX_MODULES from 1100 to 895
  (852 on 3.12, plus 5%) and adds a discover_providers() probe capped at
  916. Both counts print in the CI log.

Borrowed, not built: uv, import-linter (include_external_packages,
ignore_imports), ruff flake8-tidy-imports, deptry.

* ci: keep the bare-install smoke out of the release build job

release.yml's build job holds id-token: write and attestations: write.
Running the wheel and its ~88 transitive dependencies (resolved at
release time, not hash-locked) through validate, plan, bundle,
generate iac and doctor there widened an exposure that `fluid init`
already had. A failing doctor check could also block a tag.

A tag's merge commit has already run the same smoke on the same wheel
build in ci.yml build-check. So release.yml goes back to its
--version and init check, and the smoke runs only in CI.

* ci: the dependency budget fails loudly and ratchets

Fixes from the defect hunt on the budget guard:

- A uv failure (a resolver conflict or an index outage) now prints uv's
  own stderr. CalledProcessError's message left it out.
- `name @ url` requirements are counted. Any line the parser does not
  understand raises, so no package slips past the budget uncounted.
- --update compares against the committed baseline before rewriting it,
  so an over-ceiling run still names the packages added and removed.
- A count under its ceiling prints a notice to lower the ceiling.
- Pushes to main run the workflow only when pyproject.toml, the script,
  the baselines or the workflow change, as PRs already did. Upstream
  drift still shows in the weekly run without turning unrelated pushes
  red.

The bare-install smoke now resolves the venv path to an absolute one,
because its probe runs from /tmp, and removes its temp dir on exit.

* fix: install hints name data-product-forge, not fluid-build

Doctor, the copilot, the MCP, ODCS and DuckDB errors, the generated
Jenkinsfile and the agent docs told users to `pip install fluid-build[...]`.
The distribution is `data-product-forge`; every hint now names it with the
right extra.

- Add an empty `mcp` extra so `data-product-forge[mcp]` resolves today; it
  is filled when mcp leaves the base install.
- tests/test_no_wrong_package_name.py fails on any `fluid-build` not
  followed by `.` (domains are a separate fence). The allowlist matches a
  line, not a file (only OTel's service.namespace), and a coverage guard
  keeps the scan honest. Pattern borrowed from
  tests/test_no_reserved_tld_defaults.py.

* test: catch every shape of a wrong install hint, and missing extras

Follow-up from the security review:

- An extras-exist check: every `data-product-forge[...]` in the scanned
  files must name extras that pyproject declares. It failed first on two
  existing hints. tracing.py now names the OTel packages it imports
  (opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http), and the
  CONTRIBUTING.md example names `atlasclient`. Neither extra existed, so
  pip installed the base package and the feature still failed.
- A sentence-final "fluid-build." no longer passes as a domain.
- Install commands (pip, pip3, pipx, uv pip) fail on any fluid[-_.]build
  spelling, since PyPI normalises them to one name. The import package
  `fluid_build` stays legal everywhere else.
- The scan now covers the non-.py files shipped in the wheel and
  CONTRIBUTING.md, and asserts that it read them.
- odcs.py quotes its extra, which zsh rejected unquoted.

* fix: the vowl hint keeps its extra, and the secrets baseline follows

From the verifier:

- `fluid odcs validate` printed the vowl hint through Rich markup, which
  read `[odcs-strict]` as a tag and dropped it. Users were told to
  install the base package, which has no vowl. It now prints with
  markup=False, as generate_ci already does. A test drives the real
  cprint.
- tests/test_litellm_backend.py moved a baselined fake key from line 566
  to 607. CI's detect-secrets-hook on changed files would exit 3, so
  .secrets.baseline is refreshed. The diff is that line number and
  generated_at only; no secret entry was added.

…#789)

fluid diff --baseline read an expose's columns from exposes[].schema, but a 0.7.x
contract keeps them in exposes[].contract.schema, so a removed column, a type
change or a new NOT NULL column reported "No changes detected.", --fail-on-breaking
exited 0, and the fluid forge --refine preview said "No changes vs the existing
contract." Columns now come from contract.schema when schema is absent, and a
column is NOT NULL when required is true or nullable is false.

* fix(diff): fluid diff --baseline sees the columns of a 0.7.x contract
* fix(diff): a column is NOT NULL when required or nullable: false says so
* test(diff): reuse the shared 0.7.6 fixture and pin which spelling names a nullability finding

* feat!: retire redshift and datamesh-manager as providers

`redshift` only forwarded plan and apply to `aws`. Its real job, the
Redshift rollback DDL, is dispatched by snapshot name and never went
through the registry. Data Mesh Manager is a catalog, already reached
through `fluid publish --target datamesh-manager` and
`fluid datamesh-manager`. `fluid providers` now lists aws, databricks,
gcp, local and snowflake, plus third-party plugins.

- datamesh_manager sets __fluid_no_autoregister__, as the exporters did
  in 0.10.0. RedshiftProvider stays as the rollback-only helper.
- The error catalog and the user-facing provider lists name the real
  built-ins: no azure, and databricks added. AGENTS.md describes Data
  Mesh Manager as a catalog backend.

BREAKING CHANGE: `--provider redshift` and `--provider datamesh-manager`
(also through FLUID_PROVIDER) are no longer accepted. They exit 2 with a
one-line fix: use `--provider aws`, or `fluid publish --target
datamesh-manager`. The fix comes from one map, RETIRED_PROVIDERS, which
the global flag check, build_provider and the plan gate share. A plugin
that registers either name still wins.

* fix: reserve the retired provider names, and serve a redshift platform with aws

Fixes from the security review and the defect hunt:

- Plugins cannot claim `redshift` or `datamesh-manager`. Entry points
  load before the built-ins, so a plugin could take a retired name and
  replace its hint. Discovery now skips such a registration before
  `ep.load()`, so the plugin's code never runs, and logs a WARNING
  (`provider_retired_name_reserved`) naming the distribution. An explicit
  `register_provider(...)` call in code still works. Pattern from
  llm/provider_plugins._reserved_builtin_names.
- A contract that names `redshift` as a platform resolves to `aws` through
  RETIRED_SUCCESSORS (`builds[].execution.runtime.platform` still allows
  it). This covers resolve_provider_from_contract,
  apply._declared_platforms, policy-apply and export. So plan, apply and
  generate iac agree again, as when RedshiftProvider forwarded to
  AwsProvider. A `--provider redshift` the user types still gets the hint.
- The hint no longer needs provider discovery, so it still shows when the
  registry fails to load or an allowlist hides every provider.
- CHANGELOG: Redshift objects apply through `aws`. The rollback claim is
  dropped.

* fix: retired-name helpers pass a non-string name through

From the security re-review: resolve_retired_platform and
_claims_retired_name raised AttributeError on a non-string name, which an
LLM-drafted contract can carry. The previous commit passed such a value
through unchanged. Both now return it untouched.

* fix: the reserved-name warning names the plugin an operator must remove

The provider_retired_name_reserved WARNING carried the plugin only in a
log extra, which the CLI formatters drop. An operator saw the event but
not which package to uninstall, as two reviewers found. The message now
names the plugin and the retired name, repr'd so control characters are
escaped. The evt and plugin extras stay for log parsers.

* test: one import style per module in test_retired_providers

Code-quality review: fluid_build.providers.datamesh_manager and
fluid_build.cli were each imported with both `import` and `from ... import`
in this file. The test still checks that DataMeshManagerProvider is
exported, now through the module.

…787)

* fix(iac): lock OpenTofu state on S3 with use_lockfile

The S3 backend was emitted as bucket and key only, so a scheduled apply
and a manual one (or an apply and a teardown) wrote the same state with
no lock. tofu init now gets -backend-config=use_lockfile=true for an S3
backend when the installed OpenTofu is 1.10 or newer; the emitted module
is unchanged, so golden files stay valid. An older or unreadable
OpenTofu skips the flag and logs s3_state_unlocked with the fix, and
FLUID_TOFU_S3_LOCKFILE=0 opts out (listed in fluid doctor).

A workdir kept from before this change recorded the backend without the
lockfile, and tofu refuses the changed config ("Backend configuration
changed"). init adds -reconfigure only when the recorded backend is this
same S3 bucket and key (and region, when the module names one) without
use_lockfile; any other difference is left to tofu's own error rather than
silently repointed. Pinned by unit tests and an old-then-new apply on one
kept workdir against moto.

Upgrade note: tofu plan takes the lock too, so every role that runs plan,
apply --dry-run, fluid diff or verify --state-drift against S3 state needs
s3:GetObject, s3:PutObject and s3:DeleteObject on <key>.tflock, or sets
FLUID_TOFU_S3_LOCKFILE=0.

Found by an external hands-on assessment of 0.26.0; Trello #738 (the
locking half).

* fix(iac): the S3 lock opt-out works on a kept workdir; a timed-out tofu is interrupted first

tofu reuses the partial backend config a workdir saved whenever init gets
no -backend-config override, so once a workdir had initialised with
use_lockfile=true, FLUID_TOFU_S3_LOCKFILE=0 (or a tofu too old or
unreadable for the flag) left locking on: the operator who needed the
escape hatch stayed locked out. init now adds -reconfigure in that case,
only when the saved backend is this same bucket and key, so no state
moves.

A run that hit FLUID_TOFU_TIMEOUT_SECONDS was SIGKILLed, so tofu never
released its lock and every later run failed until someone ran tofu
force-unlock. _run now sends SIGINT, which makes tofu stop and release the
lock, and kills only after a 30 s grace period; the timeout result is
unchanged.

The lockfile-driven -reconfigure is never added with -force-copy, where
tofu would let it win and silently skip the state copy.

Found by the security review of this branch.

* test(iac): fake Popen, not subprocess.run, where tests stand in for tofu

runner._run now starts tofu with subprocess.Popen so a timeout can
interrupt it before killing it. Two tests in test_iac_state_drift.py
patched subprocess.run, so their fake was bypassed and a real Popen of
/bin/tofu failed on CI. They now patch Popen with a stand-in child.

* fix(credentials): a project .env cannot turn S3 state locking off

FLUID_TOFU_S3_LOCKFILE=0 disables a safety control, so it is classified as protected: the operator's environment may set it, a checked-in .env may not. test_every_fluid_variable_in_the_package_is_classified failed on CI because the new variable was unclassified.

* security: refuse a trailing newline in whole-string validators

Python's `$` also matches just before a final newline, so
`re.match(r"^[a-z]+$", "orders\n")` succeeds and the caller goes on to
use "orders\n". #788 closed this in providers/_sql_safety.py. The same
`^...$` plus `.match` shape guarded 36 more validator patterns across
fluid_build/, fed by contract YAML (a `|` block scalar ends in a
newline), .env values, CLI flags, LLM tool calls and percent-encoded
URL queries.

31 of those patterns now end in `\Z`. Two call sites use `fullmatch`
instead: the catalog quoting chokepoint, which also covers the BigQuery
patterns callers pass in, and the federation workspace id, whose error
message prints the pattern. Seen through the CLI before the fix,
`fluid generate iac` put `region = "eu-west-1\n"` on the AWS provider,
and `branch = "main\n"` on a Databricks job from `?branch=main%0A`.

81 other `$`-ending patterns stay as they are. Their input is stripped
first, or they are deny lists and detectors where `\Z` would loosen
the check, or they are parsers whose captured groups are what is used.

tests/security/test_trailing_newline_anchors.py pins each fixed site
with a value and the same value plus "\n". All 39 cases fail on the
parent commit.

The fix follows Django's username validators (ticket #30257, `$` to
`\Z`) and the MCP Python SDK's tool-name check (#3076, `fullmatch`).

* docs(changelog): 36 more validators refuse a value with a trailing newline

* fix(iac): refuse an AWS binding region with whitespace around it

With `\Z`, provider_block_for dropped a binding region of
"eu-west-1\n" as no region code. That left the provider on the
environment's region and skipped _guard_region_move. Pinning the
stripped value is no better. Both sovereignty checkers
(policy/sovereignty.py, providers/aws/util/sovereignty.py) compare
the region as written, so "us-east-1\n" passes
`deniedRegions: [us-east-1]` while the provider would be pinned to
us-east-1.

provider_block_for now raises UnsupportedBindingError
(aws-region-whitespace) for a region code with whitespace around it.
generate iac and apply render it as a typed CLI error before anything
is applied, and fluid diff reports the inspector error. On main the
value went onto the provider with its newline.

* security: match the BigQuery catalog and imported-schema patterns whole

copilot/catalog/bigquery.py::_validate_ident checked the same `$`
patterns with `.match` that the quoting chokepoint now checks with
`fullmatch`. So `scope.tables` values and FQN parts such as
"orders\n" reached `table_name IN ('...')`. The ODCS and ODPS
importers compile the bundled schema's identifier and semver
patterns, which end in `$`, and also used `.match`: an imported id
"orders\n" was kept verbatim, and "1.2.3\n" counted as semver. All
three now call `fullmatch`.

The two patterns that keep `$` because their call sites use
`fullmatch` (`_DEFAULT_IDENTIFIER`, `_WORKSPACE_ID_RE`) now say so
in a comment beside them.

* fix: show the refused newline in AWS name and mission path messages

These messages quoted the value as '{value}', so a name refused for a
trailing newline printed as a valid-looking name split over two
lines. They now use repr, which prints any other value exactly as
before.

* chore: keep the domain slug patterns in step with the profile-name guard

The comment on forge_domain_enrichment._SAFE_DOMAIN_RE says it
matches forge_copilot_prompts._PROFILE_NAME_RE, and the comment on
_ai_setup_storage._DOMAIN_SLUG_RE says it mirrors _SAFE_DOMAIN_RE.
The profile-name guard now ends in `\Z`, so both do too. Neither is
reachable with a trailing newline: one runs after a disk lookup, the
other on stripped input.

* docs: drop the validator count and reword validate_id's summary

The count in the changelog entry depended on how patterns were
grouped. The entry now lists what is covered instead, including the
AWS region refusal and the imports. validate_id's summary said it
refused a trailing newline "as well", which the shared grammar now
does itself.

* test: skip the BigQuery validation pin without the GCP SDK

bigquery_validation imports google.cloud.bigquery at module load. The
iac Stage 1 unit lane installs no GCP extra, so the new pin failed
collection there with ModuleNotFoundError. It now skips via
pytest.importorskip, the guard tests/providers/test_validation_sql_injection.py
already applies to the same module.

…e refuses them (#793)

* fix(iac): column types reach Glue and BigQuery faithfully or are refused at validate

fluid validate --strict exited 0 while the emitters wrote wrong or
invalid column types. Glue wrote 47 of the 79 schema types as `string`,
29 of them scalars with a real Hive type (smallint, int4, timestamptz,
real, varbinary, money ...). BigQuery got `ARRAY` / `STRUCT` with no
element type or fields through the ODCS fallback, which no BigQuery
table accepts, so the apply failed late. `array(string)` lost its
element type on both clouds.

fluid_build/iac/column_types.py classifies every type once per cloud,
and emit and validate both call it, so they cannot disagree:
- exact: the scalar maps to its faithful Glue / BigQuery type;
- lossy: a fallback the target needs (json/jsonb/variant/super/time/
  interval/bignumeric -> string on Glue; geometry -> GEOGRAPHY on
  BigQuery) is kept with a validate warning, so --strict fails;
- refused (kind column-type): array, struct, map, object, record, row and
  any parameterised structured type on both clouds (the schema cannot
  declare element types or fields yet), bit(n) with n > 1, unmapped
  types, and on Glue geography/geometry/geom/point/hll and a decimal Hive
  rejects.
The BigQuery fallback to the ODCS table is gone. _hive_type and _bq_type
stay total for fluid verify, fluid diff and the Dataform engine.

On a Glue table applied by an earlier version, the next apply changes
the column types in place (string -> smallint and so on). That update is
not a delete, so --allow-data-loss never sees it. A new optional plugin
hook, plan_update_notices(plan), lets the AWS plugin announce it on the
dry run and before the apply: one note per table, `column: old -> new`.
The CHANGELOG upgrade notes cover this, the verify/diff drift a user sees
until that apply, and every new refusal and warning.

Found by an external hands-on assessment of 0.26.0. Measured against moto
with real tofu: base-applied table, dry-run notice, apply without
--allow-data-loss, live types after.

* fix(iac): an in-place Glue column-type change needs --allow-data-loss; print plan data literally

A Glue column whose type changes in place (string -> smallint on a table
an earlier version applied) can stop Athena reading files written under
the old type until they are rewritten; reverting the type restores reads.
The AWS plugin now implements plan_data_changes: each aws_glue_catalog_
table update whose column types differ is listed, `column: old -> new`,
and the data-loss gate refuses the apply (and the dry run) until
--allow-data-loss. A whitespace-only difference (decimal(10, 2) ->
decimal(10,2)) never trips it. The advisory plan_update_notices hook from
the previous commit is gone; the gate carries the list. Like Databricks,
an AWS apply with in-place updates whose saved plan cannot be read back
now needs the flag too (fail-closed).

Plan data quotes live catalog values, so the gate list and plan notices
print through cprint_literal: a live type "string[/]" no longer crashes
every apply with a Rich MarkupError, and "[link=...]" no longer renders a
hyperlink.

Also from review: type parameters must be ASCII digits (decimal(١٠,٢) was
classified exact and emitted as written); a BigQuery decimal beyond
NUMERIC's range (scale > 9 or more than 29 integer digits) warns and
suggests bignumeric; a malformed binding is skipped by the type check
instead of raising and silently skipping the governance checks after it.

Measured against moto with real tofu: refused on dry run and apply
without the flag (live table untouched), applied with it; whitespace-only
applied without it.

…and its Lambda code takes contract values as literals (#794)

* fix(aws): the planner keeps its action order, so the emitted module is deterministic

DependencyGraph.nodes was a set, so the order of independent actions, and
with it the emitted module, followed Python's hash seed. The IaC emitter
interpolates a Lambda's attributes only when the function was emitted
first, so a Lambda permission, schedule target, event-source mapping or S3
notification named its function by literal ARN in one run and by
${aws_lambda_function.<name>.arn} in the next, and tofu sometimes had no
ordering edge to the function. nodes is now an insertion-ordered dict.

Golden check on 21 AWS contracts (examples, test fixtures, synthetic
scheduled ones), base under three hash seeds against this branch under
two: modules without a Lambda are byte-identical; scheduled ones flipped
on base and are stable here, always emitting the reference.

* fix(aws): a schedule is never dropped silently, and the placeholder it emits says so

The workflow Lambda was named fluid-workflow-<id> raw, so a dotted id (the
schema's example and 27 of 28 examples use dots) or one over ~49
characters failed the AWS plan validator, and native_actions swallowed
that refusal at DEBUG: generate iac and apply logged "Planned EventBridge
schedule" and wrote no Lambda, permission or scheduler, exit 0.

- Every AWS name the schedule planner derives from a contract id (Lambda
  functions, permission statement ids, the schedule, event rules, the
  state machine, the MWAA environment) goes through bounded_aws_name:
  disallowed characters become '-', and past the service limit the name
  is cut and ends in a hash of the whole id, so long ids stay apart. Each
  rule cites the AWS API reference; every name starts with a fixed letter
  prefix.
- A plan its own validator rejects raises PlanValidationError, which
  native_actions re-raises by type through the cause chain, like a
  sovereignty refusal: generate iac exits 1 and apply raises. A planner
  that could not run (no SDK) stays best-effort.
- An AWS account that cannot be resolved is refused by name
  (AwsAccountUnresolvedError) before any name is derived from it, instead
  of surfacing as an invalid bucket "None-fluid-staging". Offline
  generate iac keeps its placeholder account.
- The scheduled Lambda only logs its tasks and both roles it names are
  never created, so every plan that emits it warns
  schedule_placeholder_lambda, naming the roles. A real AWS scheduling
  path stays on Trello #875.

Found by an external hands-on assessment of 0.26.0; Trello #875.

* fix(aws): generated Lambda code takes contract values as literals; names never collide

From the security review of this branch:

- The workflow and trigger Lambdas' Python source was built by splicing
  contract values into string literals (TASKS = json.loads('''...'''),
  'contract_id': '<id>', FunctionName='<name>'). A schema-valid
  orchestration task param such as '''+__import__('os').system('id')+'''
  closed the literal and ran on every schedule tick under the workflow
  role; an id with a trailing newline (which the id pattern admits) made a
  Lambda that could not import. Pre-existing on main. Every contract value
  now reaches the source only as a repr() literal; tests parse each
  generated source, walk its calls and round-trip hostile ids and params.
- bounded_aws_name appends a hash of the raw name whenever sanitising
  changed it, so a.b and a-b no longer share one Lambda in an account. A
  name the service already accepts is unchanged, so nothing deployed is
  renamed.
- The orchestration and schedule-planner failure paths raised a plain
  ProviderError that native_actions swallowed, still dropping the
  schedule with exit 0; they raise PlanValidationError now.
- The S3 bucket rule allows dots as S3 does (no '..', '.-', '-.'), so a
  legal dotted bucket no longer fails generate iac now that validator
  refusals are fatal.
- An unresolved account with boto3 missing says to install
  data-product-forge[aws] instead of blaming credentials.

* test(iac): license header on the action-order test

The test file split out of test_iac_aws_schedule_names.py had no Apache header, which the License Headers check requires. Added with scripts/add_license_headers.py.

* fix(aws): an unresolved account refuses only a contract that needs the native plan

The account check ran first in AwsProvider.plan() and refused every
contract, so fluid apply of a schedule-less AWS contract failed when boto3
could not resolve the account (no [aws] extra, no STS access) although
tofu resolves it itself and main applied such contracts fine; CI's
test_state_backend_env_default.py caught it. Locally the tests passed
because this machine's credentials resolve STS.

The sovereignty check now runs first (the only place AWS enforces
jurisdiction, and it needs no account). Then, with no account, a contract
that declares orchestration (a schedule, triggers or tasks) is refused by
name, since its Lambda, permission and schedule come from no other plan;
any other contract gets a plain ProviderError, which native_actions treats
as a planner that could not run, as before.

* perf(aws): keep the new plan-error imports off the fluid --help path

provider.py is loaded by fluid --help, and its new module-scope imports pulled _plan_validation, plan.schedule and util.config onto that budgeted path. They are now imported inside the methods that use them, so build_parser() loads exactly the fluid_build modules it does on main.

…gh a pattern (#799)

* security: schema validation let a value with a trailing newline through a pattern

JSON Schema writes `pattern` in ECMA-262, where `$` matches only at the end
of the value. jsonschema runs it with Python's re.search, whose `$` also
matches before a final newline, so `fluid validate` accepted
`id: "orders\n"` (what an `id: |` block scalar produces) under every
bundled schema, 0.7.1 to 0.7.7. An editor validating the same schema in
JavaScript refuses it.

schema_manager.schema_validator(schema) extends the declared dialect's
validator with a `pattern` keyword that rewrites each `$` anchor to `\Z`.
Contract and fragment validation (_validate_with_jsonschema) and bundle
validation (forge.core.validators.validate_json_schema) use it. The
schemas are unchanged, so cached and fetched ones are covered too, and
forge_docs, the fluid spec and Command Center's models need no change.
When the line break is the only problem, the error says so.

patternProperties keys keep jsonschema's re.search, as its own
additionalProperties and _locate_error read them. No FLUID schema uses
the keyword, and a test fails if one starts to. The catalogAuth token
check, `not: {pattern: ...(?:_|$)}`, now reads `$` as ECMA-262 does
too; the name pattern beside it already refuses any value that ends in
a newline, pinned in test_catalog_auth_schema.py.

Borrowed: the keyword override from check-jsonschema
(jsonschema.validators.extend), and the `$` to `\Z` rewrite from
fastjsonschema 2.21.2 and js-regex 1.0.1. Their (?<!\\)\$ finder turns
`[$]` into `[\Z]` (bad escape) and misses `\\$`, so this scan skips
escapes and sets. Conformance case from JSON-Schema-Test-Suite
draft2020-12/optional/ecmascript-regex.json.

* security: bundle validation stays an error when jsonschema cannot be imported

Review follow-ups for the pattern end-anchor change:

- validate_json_schema checked importlib.util.find_spec, which finds a
  jsonschema that then fails to import. schema_validator then raised
  NameError, reported as a SCHEMA999 warning when not strict, where the
  base returned the SCHEMA-LIB-MISSING error. It now reads
  schema_manager.JSONSCHEMA_AVAILABLE, the flag schema_validator needs.
- The schema_validator docstring says FLUID-schema validation goes
  through it, and names the second place jsonschema's own re.search
  remains: a subschema that declares its own $schema, or a $ref to the
  root.
- CHANGELOG: a sink.catalogAuth field gets no line-break hint, since its
  value is never shown.
- Tests: the library-missing error, and a not-pattern reading $ as
  ECMA-262 does.

* test: one import style per module in the end-anchor tests

The code-quality check flagged re and fluid_build.schema_manager as imported with both import and from-import. re now comes in as `from re import sub`, and the jsonschema-missing test patches the flag by its dotted name.

…ff the clock (#800)

run_welcome_scan documented a 50ms scan that stays within _TOTAL_TIMEOUT_S,
but the budget only decided which results were kept: the ThreadPoolExecutor
context joined every probe thread. Run from a directory that holds many
repos, the specialization probe's rglob took 177s and the whole scan with it.

Each probe's wait window also started when the collector reached it, so a
cold first import counted against it. On a loaded machine the AI probe
(a lazy pydantic import inside the AI-config load) missed its window and
the scan reported "AI not configured" although it is.

- Probes run on daemon threads, one Future each. The scan returns at
  _TOTAL_TIMEOUT_S and abandons stragglers. An executor is deliberately
  not used: its non-daemon workers are joined at interpreter exit, so an
  abandoned probe would keep the CLI alive.
- _warm_probe_imports imports the modules the probes load lazily while
  the probes run, then the budget starts. A test fails if a probe imports
  something that is not listed in _WARM_MODULES.
- _PROBE_TIMEOUT_S is removed: once the probes start together it can never
  bind before the total. _TOTAL_TIMEOUT_S (1.0s) is the single budget. The
  dbt probe gets its own _DBT_VERSION_TIMEOUT_S (0.32s, was 0.4 * 0.8),
  which must stay below the budget so the child is killed by its own
  timeout rather than orphaned.
- scan_duration_ms uses a monotonic clock and covers the whole call.
- The "<50ms" claims (module docstring, forge_modes comment, CLAUDE.md) are
  corrected, and the deterministic_welcome_scan fixture drops the removed
  constant.
- tests/test_welcome_scan_budget.py pins the cap, the exit behaviour of an
  abandoned probe, the cold-import accounting, the warm list and the dbt
  cap.
- Review follow-up: _spawn_probe records only Exception; SystemExit and
  KeyboardInterrupt end the probe thread and are dropped at the deadline.

* fix(apply): refuse an unusable build id before tofu apply

`fluid apply --mode amend-and-build` (and replace-and-build) checked
contract.id and builds[].id only in the build runner. On the OpenTofu
engine the runner starts after `tofu apply`, so an id the schema admits
but the build phase refuses (1orders, more than 128 characters, a
trailing newline) was refused after the product was provisioned, and
rendered as "Unexpected error".

The runner's rule is now one function, build_runners/_ids.py
validate_build_ids, and apply runs it right after the schema gate in a
build mode, ahead of the apply hooks, tofu and the runner, on both
engines. The refusal is a typed CLIError, apply_build_identifier_invalid,
that names the field and the value and says nothing was applied. The
runner keeps its own check as the backstop.

* docs(changelog): build ids are checked before tofu apply

* test(apply): import fluid_build.cli one way in the build-id test

…ement follow-ups (#801)

* fix(databricks): the fail-safe lock never revokes from the deploy identity and says when the owner could not be skipped

* fix(databricks): the lock names a restricted table owner as a path it cannot close

* fix(databricks): the fail-safe says when the permission listing cannot show inherited grants

* fix(databricks): the pause decision is made once per run, and verify flags a job paused for an unprotected table

* fix(databricks): post_apply drops this module's superseded functions that no table in the schema uses

* fix(databricks): a residency failure prints one line with the declared region, the live region and the fix

* docs(databricks): HONESTLY_TESTED records the 2026-11-03 manual live run and what it left open

* docs(databricks): CHANGELOG entries for the masks and row filter follow-ups

* fix(databricks): the orphan drop leaves a function the deploy identity does not own

* feat(databricks): the acquisition runner lands bronze into the Unity Catalog table

A DuckDB acquisition build writing a `platform: databricks` expose no longer falls back
to a local file. The runner masks at landing as before, writes Parquet, uploads it to
DATABRICKS_STAGING_VOLUME through the Files API, loads it with one INSERT OVERWRITE ...
SELECT <contract columns> FROM read_files(...) on DATABRICKS_WAREHOUSE_ID (the table's
definition, grants, tags, row filter and masks stay), checks the row count, and deletes
the staged files whether or not the load worked. A missing table fails naming
`fluid apply`; the runner never creates or alters a table.

Because the runner applies the mask before the data leaves the machine, the emitter now
accepts policy.privacy.masking on an expose that only DuckDB acquisition builds write,
and still refuses it for dbt and every other writer.

* feat(databricks): lifecycle.expire is accepted on a full-refresh table whose Delta history fits the retention

A Databricks table rebuilt in full every run holds no row older than a run except in
Delta history, so expire is accepted when every build writing the expose is a
full-refresh writer and lifecycle.retention covers the effective history (the larger of
delta.logRetentionDuration, 30 days by default, and delta.deletedFileRetentionDuration,
7 days by default, or a dbt model's tblproperties).

Full-refresh writers: a dbt build that selects a model resolving to materialized=table
(SQL config(), then the model's YAML config, then dbt_project.yml, default view; anything
not a plain literal is refused with the fix), and a DuckDB acquisition build, whose
Databricks landing overwrites the whole table. ISO 8601 durations count years and months
at their shortest, so a too-short retention never passes. New refusals:
databricks-retention-incremental, -not-table, -dbt-unresolved, -too-short, -period;
-unmapped stays for every other writer.

fluid verify gains a retention dimension: it reads the table's properties and fails when
the live history exceeds the contract, and warns when predictive optimization is off
(VACUUM would then need scheduling).

* fix(databricks): the landing reads only its own file and deletes it on an interrupt

read_files now names the one Parquet file this run uploaded, not the run
directory, so a file someone with WRITE VOLUME drops there is not loaded.
Ctrl-C or SystemExit during the upload or the INSERT cancels the statement on
the warehouse (SqlRunner) and still deletes the staged files before the
interrupt propagates. A run killed outright cannot clean up, so each load first
deletes what earlier runs of the same build left under
<volume>/<product>/<build>/ and records it as
facets.databricks_load.earlier_runs_staging_deleted.

* fix(databricks): close the review's low findings on the landing, retention and orphan drop

- the landing's error names the files it could not delete first, so a shortened message keeps
  them, and logs them in full
- the dbt resolver does not walk a model path that reaches outside the project through a linked
  folder on the way
- the orphan drop lists tables with include_browse, so a table the run may only browse makes
  the schema unreadable (nothing dropped) instead of a binding going unseen
- verify's retention dimension warns when GetTable serves no properties, instead of passing on
  Delta's assumed defaults

* docs(databricks): say why the landing uploads with one Files API PUT

* refactor(databricks): the landing reuses the runner's path-part helper, and the orphan drop's per-function step is its own function

* fix(databricks): close the inspection's findings on the landing, retention and docs

- the INSERT OVERWRITE lists its target columns, so a live table that keeps
  its columns in another order (one fluid apply appended) gets each value in
  the column of its name, not by position
- the stale-staging sweep deletes an earlier run's directory only when every
  file in it is over an hour old, so a concurrent run keeps its upload, and it
  follows next_page_token
- retention accepts a DuckDB acquisition only with a source mode the landing
  runs (LOAD_MODES, full_refresh): another mode is databricks-retention-
  incremental, none databricks-retention-unmapped; the too-short message shows
  the declared Delta properties, not the defaults
- the dbt config scan reads Jinja tags, comments and raw blocks in linear time
  (the lazy regexes were quadratic on unclosed openers), matching them on 200k
  random inputs
- the runner and landing use public scrub/is_not_found names, and retention
  reuses util.contract.is_duckdb_acquisition_build
- a DROP FUNCTION that fails is named and the apply still succeeds (tested)
- CHANGELOG and HONESTLY_TESTED say what the code now does: masking at landing,
  orphan drop, lock identity, the second live run, counts

* fix(databricks): the masking refusal starts its reason as a sentence

* feat(databricks): an accessPolicy grant scoped by resources lands on the tables it names

The emitter refused every grant with resources (databricks-grant-scope-unmapped),
which blocked products whose grants carry $.exposes[?(@.exposeId=='<id>')].
iac/access.py now parses resources once (AccessGrant.scope, parse_grant_scope):
one expose by id (either quote, spaces allowed) or every expose ($.exposes[*],
$.exposes). The Databricks emitter grants only on the matching tables. Any other
JSONPath, alone or beside a good one, and an exposeId the contract does not have
stay refused with a message that names the resource; covers() is false for an
unevaluable scope, so a refused grant is never placed on more tables. AWS and
GCP emit what they did: GCP still applies every grant to every expose.

* fix(databricks): close the defect scan's and the second security pass's findings

- fluid validate leaves a table name, retention or source mode that is still an
  {{ env.X }} placeholder to the emitter (it refused contracts fluid apply
  accepts), and fluid verify compiles the contract with its placeholders
  substituted, as apply does; a placeholder retention promises nothing
- the emitter refuses what the DuckDB acquisition runner would refuse at run
  time after apply created the table: the source mode and sink format
  (databricks-landing-unsupported) and a masking rule it cannot apply
  (databricks-masking-invalid)
- the stale-staging sweep deletes only directories named like a run id that
  hold nothing but Parquet files, all over an hour old; an empty directory, a
  subdirectory or another file means the directory is left
- a blank accessPolicy resource is unevaluable, never "every expose"
- _contract_key returns None for a contract JSON cannot key (mixed-type or date
  keys, a cycle) instead of crashing emit, and no pause decision is reused then
- dbt fqn selectors match as dbt does: a prefix of the fqn with or without the
  project's name (shop, marts, marts.orders)
- the too-short fix rounds the history up to whole days; a COUNT that fails
  after a successful INSERT says the load ran; the lock names an owner that is
  also the run's identity once

* docs(databricks): CHANGELOG for the placeholder, landing, sweep and grant-scope fixes

* fix(databricks): match dbt selectors as dbt-core does, and refuse an unresolved landing mode at apply

- _fqn_selects ports dbt-core 1.11's is_selected_node / node_is_match (Apache-2.0):
  exact leaf name, else an exact fqn prefix up to the first wildcard part and one
  fnmatch of the rest, with and without the project's name. The per-part glob it
  replaces accepted ord*, ma*ts and *.core, which dbt does not select, so
  lifecycle.expire could be accepted for a model the build never rebuilds. Checked
  against dbt-core itself on 34,944 selector/fqn pairs.
- _refuse_unlandable skips a placeholder mode or sink format only while validating;
  at apply one still unresolved (its variable unset) is refused before the table
  is created.

* chore: refresh the secrets baseline's line numbers

* test(databricks): one blank line between the license header and the docstring, as the writer makes it

* refactor(databricks): scrub and is_not_found are the enforce module's public names; drop a pass-through lambda and repeated json imports

…'s ancestry

The previous sync (#798) was squash-merged, so b6b901f has one parent and
main never became an ancestor of feat/managed-flink-iceberg-streaming. Once
main moved on, #784 went CONFLICTING and GitHub stopped running its
pull_request workflows (CI, Flink Iceberg preview, actionlint).

This is a real merge commit of origin/main (61ccba5) into b6b901f. It
brings in #787, #793, #794, #796, #797, #799, #800, #801 and #803.

Merge base. #798's head (4d37c2c) has the same tree as b6b901f and
descends from a true merge with e59e325 (#789). So the feature's delta on
top of main is e59e325..b6b901f. git's own base, b2d05c0, predates #798's
merge. Against it, 11 files conflict, because the feature side still carries
their #789-era text. Against e59e325 the merge is clean. Outside those 11
files, both bases give the same tree. The recorded tree is the clean merge
against e59e325 (8e4c1af).

Resolutions, all 11 files git reported as conflicted:
- Took main's version, unchanged. The feature did not touch these since
  e59e325, so each conflict came from the squash alone: CHANGELOG.md,
  HONESTLY_TESTED.md, fluid_build/cli/_verify_databricks.py,
  fluid_build/iac/providers/databricks.py,
  fluid_build/iac/providers/databricks_enforce.py (add/add),
  tests/conftest.py, tests/iac/_fake_databricks.py,
  tests/iac/test_iac_databricks_enforce.py (add/add),
  tests/iac/test_iac_databricks_fake_e2e.py and
  tests/test_forge_mode_picker.py.
- fluid_build/cli/verify.py: kept both sides. Main's #801 adds the
  retention dimension and the warn status to _render_findings_dimensions.
  #784 adds --streaming-timeout and the managed-streaming branch in run().
  The hunks do not overlap.

Not conflicted:
- fluid_build/api/changelog_rules.py stays byte-identical to #798's
  judge-verified resolution. Main has not touched it since #789.
- Both sides changed these files since e59e325, and git merged them
  hunk by hunk with no overlap: README.md, build_runners/base.py,
  cli/apply.py, cli/plan.py, iac/providers/aws.py and schema_manager.py.
  _report_externally_run_build keeps #784's contract argument, and its only
  caller passes it. Main's #803 build-id gate sits after #784's early
  streaming return in apply.run(), as each side placed it.

…rry a credential to another origin (#804)

* refactor: one HTTP client in the base install, httpx

The base install carried three HTTP stacks: httpx, requests and aiohttp.
Every site now uses httpx, which is already a base dependency.

- fluid_build/util/http_client.py: new_client and new_async_client keep
  requests' behaviour (up to 30 redirects, environment proxies,
  REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE, TLS always verified). RetryTransport
  gives urllib3 Retry semantics. httpx's own transport retries cover only
  connection failures, and httpx-retries (MIT) would add a base
  dependency, so its shape is adapted here.
- Ported:
  - util/network
  - observability/reporter: refusing a foreign redirect is now a response
    hook that runs on every hop
  - cli/_command_center: HEAD does not follow redirects and GET does, as
    before
  - cli/marketplace
  - the Command Center catalog connector: aiohttp becomes
    httpx.AsyncClient with aiohttp's limits
  - the Data Mesh Manager session: GET/PUT/DELETE retried on 429/5xx with
    backoff, honouring Retry-After
  - local mocks
- util/network.safe_request reported status_code None for every 4xx and
  5xx, because the error response is falsy. It now reports the status.
- requests leaves [project] dependencies. It still installs through
  openlineage-python and tiktoken, so the package count drops when those
  leave the base install in 0.28.0.
- Tests move from requests mocks to respx and httpx. The new
  tests/test_http_client.py covers redirects, retries, proxies and TLS
  against a loopback server.

* fix: redirects are judged on the URL httpx follows, and the Data Mesh Manager key stays on its origin

Security review of the httpx port:
- refuse_foreign_redirect resolves the Location as httpx does. A Location
  with a scheme but no host (https:/x) used to pass as same-origin while
  httpx sent the key and body to the default port.
- Retry-After is capped at the 120 s backoff ceiling; an unparseable or
  overflowing value is ignored rather than crashing the CLI.
- The Data Mesh Manager session refuses a cross-origin redirect instead of
  forwarding x-api-key (older than the port); the error names the new
  origin and says to update DMM_API_URL.
- A missing REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE disables the reporter with a
  warning naming the variable; TLS is never relaxed.
- The catalog connector keeps the system trust store aiohttp used.

Drift from requests found by the defect hunt:
- a non-JSON 200 from a registry falls back again instead of crashing;
- params merge with the URL's own query string (merge_query);
- safe_request raises only on 4xx/5xx and translates allow_redirects;
- the two SSRF wiring tests fail again when the gate is removed;
- the catalog connector's search reads the real SearchFilters fields
  (broken before the port, hidden by a test shim).

* test: the redirect hook agrees with where httpx follows, for 427 Location spellings

* chore: refresh .secrets.baseline line numbers

* chore: no silent except in the ported HTTP sites

…805)

* fix(diff): validate both contracts before fluid diff --baseline compares them

The diff rules read only the shapes a schema allows, so a contract no
schema accepts could hide a change from --fail-on-breaking: a column
retyped from decimal(18,2) to the number 18 passed with exit 0.

compare_contracts now validates both sides first, as fluid plan does:
the JSON Schema of each contract's own fluidVersion, offline, from the
bundled schemas. It raises the new ContractValidationError, which lists
every error of every invalid side. A pre-0.7 contract is refused with
the upgrade fluid validate asks for. validate=False keeps the old
behaviour, and the fluid forge --refine preview uses it, because the
contract being refined may be the invalid one.

fluid diff --baseline maps the error to diff_contract_invalid, exit 2
(diff(1)'s "trouble", never 0 or 1). It logs each schema error, names
each invalid file with its fluid validate command, and prints or writes
nothing.

Prior art: datacontract-cli's breaking command parses both contracts
into typed models and refuses an invalid one (HTTP 422).

* fix(diff): carry a 0.7.x contract's quality, sensitivity and primary key

A 0.7.x contract keeps them under exposes[].contract, where the diff did
not look, so the gate missed a quality rule made failing, any sensitivity
change, and a primary-key column made nullable (reported non-breaking).

- Quality: contract.dq.rules (by id), contract.quality (by rule) and,
  from 0.7.6, each column's quality (by id) are read beside the pre-0.7
  quality.tests. An escalation is judged by fluid_build.severity.is_error,
  so critical fails as error does: warn -> critical is now an escalation
  and critical -> error no longer is.
- Sensitivity: a column whose sensitivity becomes or stops being pii or
  phi (the set docs_build and the MCP output port use) is
  column_pii_added / column_pii_removed, reported at .sensitivity. Any
  other change is the new column_sensitivity_changed (info).
- Primary key: a column tagged primary-key (or primaryKey) is in the key,
  the tag the ODCS exporter turns into primaryKey: true, so one that stops
  being required is primary_key_nullable (breaking).

The PK predicate and the sensitivity block move out of _primary_key_set
and diff_columns, which keeps both under their pre-change complexity.

* fix(diff): read the columns of a schema wrapped as {fields: [...]}

_verify_reconcile, forge_copilot_drift_guard and tools/contract_tests
accept an expose schema written as {"fields": [...]}; diff_columns read
the mapping's keys instead and saw no columns, so a removed column went
unreported. The wrapper is now unwrapped at schema and contract.schema
alike, and its columns are reported at ...schema.fields.<column>.

No bundled schema accepts the wrapper, so fluid diff --baseline refuses
such a contract as invalid; the unwrap serves the unvalidated callers,
the fluid forge --refine preview among them.

* fix(diff): print contract names as data in text, Markdown and JSON

A column or expose name reached the terminal and Markdown raw. In text
it could drive the terminal (an escape sequence, an OSC 8 link to another
URL, a carriage return or line feed that forges a line, a bidi control
that reorders it), and Rich read [bold] as a style, swapped 😄 for
an emoji and raised on [/], which crashed the command. In Markdown a
name could add a link, an image or an HTML comment that hides the rest
of a PR comment, or mention someone.

- render_text spells every non-printable or invisible character as repr
  spells it (\x1b, ‮) and doubles a backslash, and fluid diff prints
  it literally (cprint_literal) and soft-wrapped, as the drift report is,
  so each entry stays one line.
- render_markdown puts the path and the description each in one code
  span. A backslash escape is not enough: GitHub's renderer still links
  \@user, \#1, \:smile\: and foo\@example\.com (checked against its
  /markdown API); a code span is literal on every host. The fence and
  padding follow mdformat's code_inline (MIT).
- JSON and Markdown are printed verbatim (cprint_json): Rich wrapped a
  long line at 80 columns in CI, which put a line break inside a JSON
  string, so --format json could print JSON that does not parse.
- The fluid forge --refine preview prints the same text without markup.

* fix(diff): redact each value, then write JSON and Markdown verbatim

From the security review of this branch: cprint_json ran the console's
redaction regex over the finished document, and its value pattern
([^\s,;]+) took the closing fence of a Markdown code span with it. A
column named "[a](https://e.x) token=z" (or ending in token=, token:)
closed its path span early, the next fence opened the description, and
the description rendered as live Markdown: a link, an image, a mention.
The same pass took the closing quote of a JSON string, so --format json
could print JSON that does not parse.

fluid diff now redacts every value of the report first, with the
redactor logging uses (observability.secret_redactor.redact_value, a
wider set of secret shapes than the console's), and writes the JSON and
Markdown documents verbatim. No regex runs over a finished document, so
a dangling "token=" cannot reach a fence either. Text keeps
cprint_literal, whose pass cannot break a line's structure.

Tests render each name through the CLI and parse the Markdown with
markdown-it-py, and check a credential in a name or description is
redacted while the JSON still parses.

* fix(diff): refuse a contract whose fluidVersion is not a version

From the architecture review: detect_version raises on a fluidVersion
that is not a version (7, "garbage"), and the error escaped
compare_contracts' gate, so fluid diff --baseline exited 1 (diff_failed),
which a pipeline reads as "breaking". Any failure to validate a side is
now that side's error, raised as ContractValidationError, so the CLI
exits 2 (diff_contract_invalid) as for any other invalid contract.

* docs(diff): --baseline help names the validation; correct two docstrings

From the documentation review: fluid diff --help did not say that both
contracts must be schema-valid or that an invalid one exits 2 before any
output, which only the CHANGELOG said. The --baseline help says so now.
The _code docstring states its padding rule as the code applies it, and
the module docstring no longer places the renderers in cli/diff.py.

* fix(diff): a duplicate rule id cannot hide a severity escalation

From the codebase review: nothing makes a quality rule's id unique, and
_rules_by_name kept the last rule of each name, so a new contract with
dq.rules [d1 error, d1 warn] against a baseline [d1 warn] reported no
escalation, and --fail-on-breaking passed. A baseline [d1 error, d1 warn]
against [d1 error, d1 error] was reported as one.

Each name now keeps all of its rules, and each version is judged by its
most severe rule of that name, in every location the diff reads.

* test(diff): pin the code-span padding and the defaults the review found untested

From the test-suite review (mutation probes): the space-at-both-ends and
backtick-at-both-ends cases embedded the name in a longer path, so they
never reached _code's padding branch, and a mutant that dropped it
survived. _code is now tested on the raw strings, parsed with
markdown-it-py. Also pinned: a quality test with no severity counts as
warn; a column rule named by name (no id) is compared; an invalid
baseline alone is named in diff_contract_invalid; the pre-0.7 per-column
primaryKey flag is still read.

* fix(diff): the defect scan's findings in the diff engine

From the scoped defect scan of this branch (hunters, skeptic, referee):

- A read-only or missing HOME made fluid diff --baseline exit 1, as for a
  breaking change: the schema manager creates ~/.fluid/schema_cache when
  it is built. It now falls back to a temporary cache; the diff validates
  offline against the bundled schemas, so nothing a bundled version needs
  is lost.
- A column the dbt engine builds as a key (engines/dbt/_test_mapping.py
  column_is_key: semanticType identifier or primary_key, a labels
  constraint of primary_key or unique, the key flags) is in the primary
  key, as a primary-key tag already was, so one that stops being required
  is primary_key_nullable. The predicate is mirrored, because importing
  the engine loads every engine; a test pins the two together.
- A PII change reported at .sensitivity carries the sensitivity values,
  not the derived booleans, and a pre-0.7 pii: true rewritten as
  sensitivity: pii is not a change.
- The blank Braille pattern (U+2800), a printable symbol that draws as a
  space, is spelled out like the other characters that draw nothing.

* fix(diff): one redacted, JSON-safe envelope for stdout and --out

From the scoped defect scan of this branch:

- The per-value redaction used the logging redactor alone, which misses
  shapes the console pattern caught on main: "api key: x" with a space,
  a separator padded with more than eight spaces, a very long value. A
  value now goes through both, as the CLI's error renderer does
  (_redact_str(redact_secret_text(...))).
- --out was written from the raw report while stdout was redacted. Both
  now get the same redacted envelope (_safe_report).
- A value JSON cannot hold (a date, bytes or a set that YAML put in a
  column's quality rule) made json.dump raise after the text report had
  printed, left --out cut short and exited 1, as for a breaking change; a
  NaN was written as a bare token. Such values are written as text.

* refactor(diff): judge a key with the dbt engine's own predicate, not a copy

The diff mirrored engines/dbt/_test_mapping.py::column_is_key (and its
is_truthy) because importing the engine loads every engine; ripwire's
quality delta flagged the 141-token copy. column_is_key, is_truthy and
_PK_KEYS move unchanged to fluid_build/util/contract.py (stdlib-only),
_test_mapping re-exports them so every _tm.* caller is unchanged, and
_in_primary_key is the ODCS tag check plus that one predicate. The
parity test becomes an identity check.

* fix(diff): redact a credential held in a mapping key, too

From the round-2 review: _safe_report redacted every string value, but a
mapping's keys went out as written, and a removed column's free-form
labels reach before/after. A key such as "password: x" was printed in
--format json and written to --out; on main the whole-text pass of the
console had masked it on stdout. Keys now go through both redactors.

… are reserved behind an operator grant (#807)

* feat(iac): IaC plugins load on first use, and built-in platform names are reserved

A fluid_build.iac_providers entry point is no longer imported with
fluid_build.iac. Its name is read with importlib.metadata, and the plugin
is loaded, by name, when a contract's platform or --provider needs it
(the stevedore DriverManager shape). The allow/block lists and
requires_cli are checked before the load.

An entry point that claims a built-in platform or alias is skipped before
import, with a WARNING naming the distribution. The operator can grant
an override with a distribution-qualified FLUID_PLUGINS_ALLOWLIST entry
(<dist>:<name>, plugin_manager.override_granted). Retired names are
never granted.

* fix(iac): grants match the real distribution name, policy matches every spelling, and the state guard sees unloaded plugins

Security review and defect hunt on the lazy IaC plugin registry:
- override_granted reads ep.dist.name (PEP 508 fullmatch) instead of the
  first word of "<name> <version>", so a self-declared "acme-fluid-aws
  spoof" no longer satisfies the grant acme-fluid-aws:aws.
- Allow/block entries match case-insensitively with - == _, so an entry
  point named Kubernetes or on-prem cannot slip past a blocklist written
  kubernetes or on_prem; policy, reserved names and lookup share one key.
- The state-sharing guard loads every admitted plugin before attributing
  state, so an --allow-data-loss apply on a shared key still sees a
  plugin cloud's resources that nothing loaded earlier in the process.
- Distribution and entry-point names are repr'd in the reserved-name
  hint and the unavailable reason, so a name cannot inject escapes.
- A grant cancelled by a bare allowlist says so; two distributions
  shipping one platform name are refused rather than last-one-wins.
- get_iac_plugin re-checks under the lock (one instantiation per
  platform) and is stable for any spelling; reset restores a built-in
  that a grant replaced.
- Docs: requires_cli is checked right after the import, and the global
  --provider flag accepts built-ins only (auto-detect reaches plugins).

* chore(iac): assign the entry-point index cache on its own line

… extras ship alongside the base install (#806)

* feat: optional dependencies name their extra, and silent fallbacks warn

Step 5 of the dependency diet. Every optional-dependency failure now says
what to install, in one line, and exits 1 with no traceback. Nothing
changes when everything is installed.

- fluid_build/_optional.py adds require(module, extra, feature) and
  MissingOptionalDependency, a subclass of ImportError. It is adapted
  from pandas' import_optional_dependency (BSD-3).
  - It probes with find_spec before importing, top-level package first,
    since find_spec on a dotted name raises when the parent is missing
    (bpo-30436).
  - A package that is installed but fails to import keeps its own error.
  - The CLI prints the message with its stable code and exits 1.
- These now go through it: ai setup's key validation, the litellm
  backend, LLM fallback routing, the MCP server and output-port server,
  the dbt, hosted and discovery MCP delegates, and fetch_sample_rows
  (duckdb).
- fluid doctor never imports litellm, mcp or openlineage; it reads their
  versions from package metadata. That takes it from 3,287 to 1,035
  modules and drops litellm's cost-map fetch. New MCP SDK and OpenLineage
  rows name their extras.
- AWS: plan and diff used to fail with "Bucket name 'None-fluid-staging'
  is invalid". They now name the [aws] extra when boto3 is missing, and
  say "Could not resolve the AWS account id" when no credentials resolve.
  Glue import discovery warns instead of silently dropping its blocks.
- Lineage emission without openlineage-python warns once, naming
  [lineage], and apply still succeeds. The ODCS and credential fallbacks
  warn instead of logging at DEBUG.
- _mcp_compat probes mcp before mcp.server.mcpserver.
- Corrected comments that called litellm a hard dependency, or said
  sqlglot powers _sql_safety.
- An empty `lineage` extra lets the [lineage] hint resolve now; the
  extras PR fills it.
- .secrets.baseline: one entry's line number moved.

* fix: the lineage warning keeps its [lineage] extra on the console

From the security review of the optional-dependency commit:

- _warn_once printed through _console.warning, which wraps the text in
  Rich markup. Rich read `[lineage]` as a tag and dropped it, so the
  console said `pip install 'data-product-forge'`. It now prints with
  literal_text. The test asserts the console copy as well as the log
  record; it failed before this change.
- The row_access comment claimed the separator and comment check is
  skipped without sqlglot. That check runs first and still holds. Only
  the structural checks (subquery, UNION/GROUP/ORDER/LIMIT) and the
  undeclared-column check go unchecked.

* feat: litellm, mcp and lineage extras alongside the base install

0.27.0 adds the extras that 0.28.0 relies on when litellm, mcp and
openlineage leave the base install. The base install is unchanged here;
a package listed in both is harmless to pip and uv.

- [litellm] carries both base lines verbatim, keeping the Python 3.10
  cap from PR #555, and [ai] aliases it.
- [mcp] is mcp plus pyjwt, which output_ports/mcp/auth.py imports
  directly.
- [lineage] is openlineage-python.
- [standard] is litellm + mcp, as in fastapi[standard].
- all and dev include the new extras.
- CI jobs that run LLM or MCP code now name their extras (llm-smoke,
  ai-provider-matrix, four integration jobs), so 0.28.0 changes nothing
  for them.
- Every Docker profile installs [standard], so the images keep AI and
  `fluid mcp serve`.
- `fluid doctor` prints one line: AI/MCP move to extras in 0.28.0.
- mcp_output_port_certify.py names [mcp] when the SDK is missing.
- tests/ci_guards/extras_alongside_base.py pins the marker lines and
  the contents of [standard], [ai], all and dev.

* fix: pyjwt floor 2.15.0 in [mcp], and Docker images keep lineage

From the security review of the extras commit:

- [mcp] declared pyjwt>=2.10.1, mcp's own floor. Every release up to
  2.14.0 carries advisories on the PyJWKClient and decode path that
  output_ports/mcp/auth.py uses (e.g. GHSA-ffc3-869f-jxw9, critical).
  2.15.0 has none. The floor is now pyjwt[crypto]>=2.15.0, which
  resolves on 3.10 and 3.12 alongside every mcp in range.
- openlineage-python leaves the base install in 0.28.0, but
  [standard] is only litellm + mcp. Every Docker profile now names
  [lineage] too, so the images keep OpenLineage emission. Today's
  image is unchanged, since all of it is still in base.

* chore: keep the real [lineage] extra, drop the empty alias

* fix: install hints survive every output path, and a broken litellm degrades like a missing one

From the defect hunt:
- fluid forge without litellm printed the hint as Rich markup, which
  swallowed [litellm], and dumped tracebacks; the hint is now literal
  and a missing optional dependency logs no traceback.
- An installed litellm that fails to import (>= 1.98 on Python 3.10)
  escaped list_available_models as a raw ImportError and broke the
  model preflight; it is a CopilotGenerationError again.
- doctor probed the openlineage namespace package, so any openlineage-*
  distribution read as installed; it probes openlineage.client.
- The dbt and hosted MCP delegates logged only the exception type, so
  the [mcp] hint never reached the user; a MissingOptionalDependency
  message (no secrets in it) now goes through.
- The lineage warning keeps the pip line on its own line.
- The AWS account-id error no longer claims there are no credentials
  when STS failed for another reason.
- MissingOptionalDependency pickles and copies.
- A test left a stub output-port module in sys.modules.

* chore: one blank line after the license header, as the header writer emits

* docs(changelog): the extras and the optional-dependency gate

* fix: a failed mcp serve leaves no policy behind, and AWS account errors are main's

- fluid mcp serve installed its process-wide policy before building the
  app, so a missing SDK left a policy with another directory's readable
  paths installed; in-process MCP tool calls after it were refused. The
  app is built first now.
- The serve test clears the cached app, which an earlier in-process test
  could have built, so it exercises the SDK import every time.
- The AWS account-id guard added on this branch is gone: main now
  reports an unresolved account itself (#794), after the sovereignty
  check and with the [aws] hint, and the guard pre-empted it.

* test: one import style per module in the optional-deps tests

* test: run the no-SDK output-port check in a subprocess, and reset the mcp version cache around blocked imports

* chore: refresh .secrets.baseline line numbers

Main moved while the first sync merge (6d34df5) was being checked: #804,
#805, #806 and #807 landed. Without them, #805 conflicts with #784 in
fluid_build/cli/diff.py and #784 stays CONFLICTING. This is a normal merge;
its base is 61ccba5, which is now a real ancestor.

Resolution, one conflict:
- fluid_build/cli/diff.py, imports. Kept main's
  `from ._logging import error, info, warn` (#805's schema-error reporting
  uses `error`) and #784's `from .console import cprint` (the managed Flink
  drift summary uses it).

Both sides also changed these files since 61ccba5; git merged them hunk by
hunk with no conflict:
- fluid_build/api/changelog_rules.py:
  - #805 replaces the PII drift with `_sensitivity_changes`, reads a
    `schema: {fields: [...]}` wrapper in `_expose_schema` and adds the
    sovereignty and quality-severity rules.
  - #784's container-child recursion and nested nullability stay.
  - Every function except `diff_columns` equals the version from the side
    that changed it (AST comparison against 61ccba5).
  - `diff_columns` runs #784's `_diff_container_children` call, then #805's
    `_sensitivity_changes`.
- fluid_build/iac/providers/aws.py: #806's account-id warnings in
  `_resolve_catalog_id`. #784's nested `_columns` branch is unchanged.

The tree differs from origin/main (ef28db6) in exactly the feature's 39
files, and from 6d34df5 in exactly main's 88 files.

`fluid apply --mode amend-and-build --build-id <id>` (and replace-and-build) checked the id only in the build runner, which keeps the builds whose id equals it and, with none left, logs "Build not found" and returns 1. On the OpenTofu engine the runner starts after `tofu apply`, so a mistyped id provisioned the product and then failed.

apply now runs the runner's match in a build mode, right after the build-id grammar gate, ahead of the apply hooks, tofu and the runner, on both engines. The refusal is a typed CLIError, apply_build_id_not_found, that names the id, lists the contract's build ids and says nothing was applied. Ids match exactly, as in the runner. An empty --build-id and a contract without builds are not checked, as before. The runner keeps its own check as the backstop. The Command Center run now reports apply_build_id_not_found, where it reported build_not_found:<id> after the infrastructure was applied.

* docs(changelog): an unknown --build-id is refused before tofu apply

The feature branch moved while this sync was in review: #809 (the security
gate) was squash-merged onto b6b901f as f4768da. Merging it here keeps
this sync mergeable into the feature branch. The base is b6b901f.

Resolution, one conflict:
- fluid_build/cli/diff.py, imports. Kept main's `_redact_str` and
  `redact_secret_text` (#805's redacted schema-error output) and #809's
  `FluidUserError`, which `run()` now re-raises as a typed error.

pyproject.toml merged without conflict. #809 adds its three modules to the
mypy strict allowlist, and main's #806 extras stay.

The tree differs from f4768da in exactly main's files, and from 95c5c1c
in exactly #809's 12 files. In the two files both sides changed, each
side's diff is unchanged.

Brings in #808, which landed during review. The merge is clean, with base
ef28db6.

#808 and #784 both change two files:
- fluid_build/cli/apply.py. #808's `_gate_selected_build_for_apply` runs in
  the general gate block after `_gate_build_ids_for_apply`. Like #803's gate,
  that is after #784's managed-streaming early return, which refuses any
  --build-id on its own (`apply_streaming_build_selection`).
- fluid_build/build_runners/base.py. #808 changes only a comment in the
  build filter; #784's `_report_externally_run_build` is unchanged.

…e=False

#805 made compare_contracts validate both contracts by default. This test's
contract is schema-invalid on purpose (a `name` inside a map value), so it
now raised ContractValidationError before the rule under test ran. The test
checks the diff rule, not validity, so it opts out with validate=False, the
path #805 keeps for a contract that is still being repaired.